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Abstract. We show in a dynamic programming framework that uniform convergence of the 
finite horizon values implies that asymptotically the average accumulated payoff is constant on 
optimal trajectories. We analyze and discuss several possible extensions to two-person games. 



1. Presentation 

Consider a dynamic programming problem as described in Lehrer and Sorin pQ. Given a set of 
states S, a correspondence 3> from S to itself with non empty values and a payoff function / from 
S to [0, 1], a feasible play at s € S is a sequence {s m } of states with s± = s and s m+ i E <3?(s m ). It 
induces a sequence of payoffs {f m = f(s m )}, m = 1, n, .... Recall that starting from a standard 
problem with random transitions and/or signals on the state, this presentation amounts to work 
on the set of probabilities on S and to consider expected payoffs. 

Let v n (s) (resp. v\(s)) be the value of the n stage program G n (s) (resp. A discounted program 
G\(s)) starting from state s. The asymptotic approach deals with asymptotic properties of 
the values v n and v\ as n goes to oo or A goes to 0. 

The uniform approach focuses on properties of the strategies that hold uniformly in long 
horizons. is the uniform value if for each e > there exists N such that for each s G S: 
1) there is a feasible play {s m } at s with 



-J2f{s m )>Voo(s)-e, Vn>N 

m=l 

2) for any feasible play {s' m } at s and any n > N 

1 - 

- Yl ^ s '^> - v oo(s) + e. 

m=l 

Obviously the second approach is more powerful than the second (existence of a uniform value 
implies existence of an asymptotic value : the limit of v n exists) but it is also more demanding: 
there are problems without uniform value where the asymptotic value exists (see Section 2). 
Note that the condition for the existence of a uniform value implies that the average accumulated 
payoff on optimal trajectories remains close to the value. 

We will prove that a similar phenomenon holds true under conditions that are stronger than 
the existence of an asymptotic value but weaker than the existence of a uniform value. 

Say that the dynamic programming problem is regular if : 

i) limv n (s) = v(s) exists for each s € S. 

ii) the convergence is uniform. 

This condition was already introduced and studied in Lehrer and Sorin [1] (see Section 2). 
We consider the following property P: 



Date: October 2009. 



1 



2 



SYLVAIN SORIN, XAVIER VENEL, GUILLAUME VIGERAL 



For any e > 0, there exists no, such that for all n > uq, for any state s and any feasible play 
{s m } e-optimal for G n (s) and for any t € [0, 1]: 

[tn] 

(1) 3e>-(V/ m )-to(s)>-3£. 

n 

m=l 

where [tn] stands for the integer part of tn. 

This condition says that the average payoff remains close to the value on every almost-optimal 
trajectory with long duration (but the trajectory may depend on this duration). 
It also implies a similar property on every time interval. 



2. Examples and comments 

1) The existence of the asymptotic value is not enough to control the payoff as required in 
property P. 

An example is given in Lehrer and Sorin pQ (Section 2), where both \\mv n and \imv\ exist on 
S but where the asymptotic average payoff is not constant on the unique optimal trajectory, nor 
on e-optimal trajectories: in Gm^ an optimal play will induce n times then n times 1 while 
v = 1/2. 

Note that this example is not regular: the convergence of v n to v is not uniform. 

2) Recall that in the framework of dynamic programming, regularity is also equivalent to 
uniform convergence of v\ (and with the same limit), see Lehrer and Sorin [T] (Section 3). 
Note also that this regularity condition is not sufficient to obtain the existence of a uniform value, 
see Monderer and Sorin [2] (Section 2). 

3) General conditions for regularity can be found in Renault [5J. 

3. Main result 
Theorem 3.1. Assume that the program is regular, then P holds. 

Proof 

Let us start with the upper bound inequality in JT]). 
The result is clear for t < e (recall that that the payoff is in [0, 1]). Otherwise let n\ large enough 
so that n>n\ implies ||t? n — v|| < e by uniform convergence. Then the required inequality holds 
for n > ri2 with [en-z\ > n\. 

Consider now the lower bound inequality in ([1]). 
The result holds for t > 1 — e by the e-optimal property of the play, for n > n\. Otherwise we 
use the following lemma from Lehrer and Sorin [1] (Proposition 1). 

Lemma 3.1. Both lim sup v n and \imsupv\ decresase on feasible histories. 

In particular, starting from sr tn i the value of the program for the last n — [tn] stages is at most 
v ( s [tn]) + e for n > n-2, by uniform convergence, hence less than the initial v(s) + e, using the 
previous Lemma. Since the play is e-optimal in G n (s), this implies that 

[tn] 

(2) Yl fm + {n ~ [tn])(v(s) + e) > n(v n (s) - e) > n(v(s) - 2e) 

m=l 

hence the required inequality. ■ 

4. Extensions 

4.1. Discounted case. 

A similar result holds for the program G\ corresponding to the evaluation Ylm=i -^(1 — ty m fm- 
Explicitly, one introduces the property P': 
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For any e > 0, there exists Ao, such that for all A < Ao, for any state s and any feasible play {s m } 
e-optimal for G\(s) and for any t € [0, 1]: 

n(t;A) 

(3) 3e > A (! " A) m -V m ) - tv(s) > -3s. 

m=l 

where n(t; A) = inf{p € M; Ylm=l A (l ~~ A) m > i}. Stage n(i; A) corresponds to the fraction t 
of the total duration of the program. 

Theorem 4.1. Assume that the program is regular, then P' holds. 
Proof 

The proof follows the same lines than the proof of Theorem 13.11 
Recall that by regularity both v n and v\ converge uniformly to v. Moreover the discounted sums 
(1 - \)~ N Y,m=i A (! - A ) m_1 /m belong to the convex hull of the averages ± £m=i fm\ 1 < n < N. 
The counterpart of equation ([2]) is now 

n(t;X) 

(4) J2 A (! " A ) m "Vm + (1 - t)(v(s) + e)> (v x (s) -e)> v(s) - 2e 

m=l 



4.2. Continuous time. 

Similar results holds in the following set-up: vt(x) is the value of the control problem Tt with 
control set U where the state variable in X is governed by a differential equation (or more generally 
a differential inclusion) 

xt = f(x t ,u t ) 

starting from x at time 0. The real payoff function is g(x, u) and the evaluation is given by: 

T 

g(x t ,u t )dt. 

Regularity in this framework amounts to uniform convergence (on X) of Vt to some V . (Sufficient 
conditions for regularity can be found in Quincampoix and Renault [4J). The corresponding 
property is now P": 

For any e > 0, there exists To, such that for all T > To, for any state x and any feasible trajectory 
e-optimal for Tt(x) and for any 9 € [0,1]: 

1 f 0T 

(5) 3e > - / g(x t ,u t )dt - 6V{x) > -3e. 

1 Jo 

Theorem 4.2. Assume that the optimal control problem is regular, then P" holds. 
Proof 

Follow exactly the same lines than the proof of Theorem ([2]). ■ 
Finally the same tools can be used for an evaluation of the form A / + °° e ~ Xt 9( x t,ut)dt. 




5. Two-player zero-sum games 
In trying to extend this result to a two-person zero-sum framework, several problems occurs. 
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5.1. Optimal strategies on both sides. 

First it is necessary, to obtain good properties on the trajectory, to ask for optimality on both 
sides. 

For example in the Big Match with no signals, 



where a * denotes an absorbing payoff, the optimal strategy of player 1 in the "asymptotic game" 
on [0, 1] is to play "a before time t" with probability t, see Sorin [6] Section 5.3.2. Obviously, 
if there is no restrictions on player 2's moves the average payoff will not be constant. However, 
the optimal strategy of player 2 is "always (1/2, 1/2)" hence time independent on [0, 1]. It thus 
induces a constant payoff and it is easy to see that the property is robust to small perturbations 
in the evaluation of the payoff. 

5.2. Player 1 controls the transition. 

Consider a repeated game with finite characteristics (states, moves, signals, ...) and use the 
recursive formula corresponding to the canonical representation with entrance laws being consis- 
tent probabilities on the universal belief space, see Mertens, Sorin and Zamir [3], Chapters III. 1 , 
IV. 3. This representation preserves the values but in the auxiliary game, if player 1 controls the 
transition an optimal strategy of player 2 is to play a stage by stage best reply. Hence the model 
reduces to the dynamic programming framework and the results of the previous sections apply. 
A simple example corresponds to a game with incomplete information on one side where asymp- 
totically an optimal strategy of the uniform player 1 is a splitting at time 0, while player 2 can 
obain u{pt) at time t where u is the value of the non-revealing game and pt the martingale of 
posteriors at time t, see Sorin [6], 3.7.2. 

5.3. Example. 

Back to the general framework of two person zero-sum repeated games, the following example 
shows that in addition one has to strengthen the conditions on the pair of e-optimal strategies. 
We exhibit a game having a uniform value v but for some state s with v(s) = one can construct, 
for each n, optimal strategies in T n (s) inducing essentially a constant payoff 1 during the first 
half of the game. 

Starting from the initial state s, the tree representing the game T has countably many subgames 
T2n, the transition being controlled by player 1 (with payoff 0). In T2 n there are at most n stages 
before reaching an absorbing state. At each of these stages of the form (2n,m),m = l,...n, 
the players plays a "jointly controlled" process leading either to a payoff 1 and the next stage 
(2n, m + 1) (if they agree) or an absorbing payoff X2 n ,m with (m — 1) + (2n — (m — l))x2 n ,m = 0> 
otherwise. Hence every feasible path of length 2n in T2 n gives a total payoff 0. Obviously 
the uniform value exists since each player can stop the game at each node, inducing the same 
absorbing payoff. The representation is as follows: 

Notice that in the 2n + 1 stage game, after a move of player 1 to T2 n , any play is compatible 
with optimal strategies, in particular those leading to the sequence of payoffs 2n times or n 
times 1 then n times —1. 

5.4. Conjectures. 

A natural conjecture is that in any regular game (i.e. where v n converges uniformly to v): 

for any e > 0, there exists no, such that for all n > no, for any initial state s, there exists a couple 

(cn> T n) °f e-optimal strategies in G n (s) such that for any t € [0, 1]: 



a (3 



a 1* 0* 



b 1 




[in] 



(6) 



m=l 
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Figure 1. The game T starting from state s 
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Figure 2. The subgame T2n starting from state (2n, 1) 



where [tn] stands for the integer part of tn and f m is the payoff at stage m. 
A more elaborate conjecture would rely on the existence of an asymptotic game T* played in con- 
tinuous time on [0, 1] with value v (as in Section 5.1). We use the representation of the repeated 
game as a stochastic game trough the recursive structure as above, see Mertens, Sorin, Zamir [3], 
Chapter IV. The condition is now the existence of a couple of strategies (<r, r) in the asymptotic 
game that would depend only on the time t G [0, 1] and on the current state s such that for 
any e > 0, there exists r/ with the following property: in any repeated game where the (relative) 
weight of stage m is a m , with {<x m } decreasing and less than rj, thus defining a partition II of 
[0, 1], the strategies (crn^n) induced in the repeated game by (a, r) satisfies ©. 
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